A [[Sparsity x LLMs]] approach from the HazyResearch lab at Stanford.
- The Pixelated Butterfly method **operates at the level of matrix operations**, which are fundamental to many AI models. Matrix operations involve multiplying matrices, which are arrays of numbers, to perform computations.
- The authors of the Pixelated Butterfly method combine **specialized butterfly** and **low-rank matrices** to create a simple and efficient sparse training method.
- Butterfly matrices are a type of matrix that can be used to **approximate other matrices**,
- Low-rank matrices have a small number of non-zero elements.
- By using butterfly and low-rank matrices, the Pixelated Butterfly method can effectively **reduce the number of non-zero elements in a matrix**, which in turn reduces the computational complexity of the AI model.
- The Pixelated Butterfly method **can be applied to most major network layers** that rely on matrix multiplication, which includes many popular types of neural networks.
- Overall, the Pix elated Butterfly method is an example of how sparsity techniques can be used to simplify AI models and make them more efficient.
![[Pasted image 20230801120624.png]]
![[Pasted image 20230801120500.png]]
Ref:
1. https://hazyresearch.stanford.edu/blog/2022-01-17-Sparsity-3-Pixelated-Butterfly
2. https://arxiv.org/abs/2112.00029